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USER INTERACTIVE LEVEL SET METHODS FOR IMAGE SEGMENTATION 

This application claims the benefit of U.S. Provisional 

Application No. 60/431,367, filed December 6, 2002, and U.S. 

Provisional Application No. 60/520,450, filed November 14, 

2003 entitled ''User-Aided Boundary Delineation Through the 

Propagation of Implicit Representation", which are 

incorporated by reference herein in their entirety. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention: 

The present invention relates to image segmentation, 
and more particularly to a system and method for image 
segmentation implementing an interactive level set. 

2 . Discussion of Related Art : 

Image segmentation approaches can be classified as 
boundary or region-based. Boundary- driven techniques rely 
on the generation of a strength image and the extraction of 
prominent edges, while region-based methods rely on the 
homogeneity of spatially localized features and properties. 
Snake -driven techniques are typically the most appropriate 
tool to derive boundary-based methods. A curve propagation 
technique is a common way to implement such terms. 
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To this end, a parameter space that defines a curve in 
the image plane is considered. Object extraction is 
equivalent with finding the lowest potential of an objective 
function. Such a function involves internal and external 
5 terms. The internal term enforces some desired geometric 
characteristics of the curve, while the external term moves 
the curve to the desired image features. Level set methods 
address such an objective in various application domains. 

Level set formulations consider the problem in a higher 

10 dimension and represent the evolving curve as the zero-level 
set of an embedding function. The evolution of this 
function can then be derived in a straightforward manner 
from the original flow that guides the propagation of the 
curve. Such methods are implicit, intrinsic and topology 

15 free leading to a natural handling of important shape 
deformations . 

An important limitation of level set formulations is 
sensitivity to noise and failing to capture/encode prior 
knowledge shape-driven on the structure to be recovered. A 
20 geometric flow that evolves the solution closer to the prior 
can introduce prior shape knowledge within the segmentation 
process. A more elegant formulation was derived in which 
such constraints were introduced in the form of energy 
components that constrain the solution space. 
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User-interaction is an important component in medical 
segmentation where boundary-tracing tools are implemented. 
User interaction, can be considered as a different form of 
prior knowledge to be added in the segmentation process. 
5 Recent advances in medical imaging have increased the 

accuracy of automated techniques. However, clinical users 
typically need to correct their outcome. Although, level 
set methods are an established segmentation technique in 
medical imaging they do not support user interaction. 

10 

SUMMZ^Y OF THE INVENTION 

According to an embodiment of the present invention, a 
method for boundary based image segmentation comprises 
segmenting an image, providing a level set representation of 

15 the segmentation for interaction, and providing an edit 

point of the level set representation. The method further 
comprises converting the edit point into a propagation 
constraint, and determining a segment according to the edit 
point and the level set representation. 

20 The edit point is one of a control point and a sequence 

of connected points. 

Converting further comprises deriving a shape 
constraint within a level set framework wherein an 
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interpolation converts the edit point into a closed 
structure. 

The method comprises enforcing a smoothness constraint 
on the level set representation to correct a local 
5 discrepancy given the control point. The method comprises 
replacing a segment of the level set representation with an 
interaction segment according to the control point. The 
segment of the level set representation is determined to lie 
within a predetermined distance from the control point. 

10 The method comprises introducing the interactive segment, 

wherein the interactive segment is determined by a quadratic 
interpolation between the control point and the segment of 
the level set representation. 

The sequence of connected points provides a number of 

15 control points in a clock-wise order that when connected 
define a closed curve. The method comprises recovering a 
global constraint that forces the level set representation 
to go through the number of control points. 

The method comprises evolving the level set 

20 representation according to the propagation constraint. The 
method further comprises minimizing a distance between the 
evolving level set representation and the propagation 
constraint . 
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The propagation constraint is enforced more stringently 
closer to the control point. 

The method comprises evolving the level set 
representation locally towards the propagation constraint. 
5 According to an embodiment of the present invention, a 

program storage device is provided readable by machine, 
tangibly embodying a program of instructions executable by 
the machine to perform method steps for boundary based image 
segmentation. The method steps comprising segmenting an 

10 image, providing a level set representation of the 

segmentation for interaction, and providing an edit point of 
the level set representation. The method further comprises 
converting the edit point into a propagation constraint, and 
determining a segment according to the edit point and the 

15 level set representation. 

According to an embodiment of the present invention, a 
method for image segmentation comprises providing a level 
set representation of an image segmentation for interaction, 
requesting user-interaction for editing the level set 

20 representation, determining a propagation constraint 
according to the user-interaction, and determining an 
evolved level set representation, wherein a portion of the 
evolved level set representation is locally affected by the 
propagation constraint. 
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The user-interaction is one of a control point and a 
sequence of connected points. The method further comprising 
enforcing a smoothness constraint on the level set 
representation to correct a local discrepancy given the 
5 user- interact ion . 

The portion of the evolved level set representation 
replaces a corresponding portion of the level set 
representation. The portion segment of the level set 
representation is determined to lie within a predetermined 
10 distance from the user-interaction. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Preferred embodiments of the present invention will be 

described below in more detail, with reference to the 
15 accompanying drawings: 

Figure 1 is a diagram of a system according to an 

embodiment of the present invention; 

Figure 2 is an illustration of level set 

representations and evolving surfaces according to an 
20 embodiment of the present inventions- 
Figures 3a-3b are segmentation images according to an 

embodiment of the present inventions- 
Figure 4 is a graph of a quadratic interpolation 

according to an embodiment of the present invention; and 
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Figure 5 is a flow chart of a method according to an 
embodiment of the present invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

5 According to an embodiment of the present invention, 

user-defined segmentation constraints can be ' implemented 
within level set methods. Snake-driven methods are powerful 
and widely explored techniques for object extraction- A 
level set representation is a mathematical framework 

10 technique to implement snake-driven methods. This 

formulation is implicit, intrinsic and parameter/topology 
free. User interactive constraints are a form of prior 
shape knowledge. To this end, a formulation is implemented 
that converts user interaction to objective function terms 

15 that guide and improve the segmentation through the user 
edits. 

It is to be understood that the present invention may 
be implemented in various forms of hardware, software, 
firmware, special purpose processors, or a combination 
20 thereof. In one embodiment, the present invention may be 
implemented in software as an application program tangibly 
embodied on a program storage device. The application 
program may be uploaded to, and executed by, a machine 
comprising any suitable architecture. 
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Referring to Fig. 1, according to an embodiment of the 
present invention, a computer system 101 for implementing 
the present invention can comprise, inter alia, a central 
processing unit (CPU) 102, a memory 103, and an input/output 

5 (I/O) interface 104. The computer system 101 is generally 
coupled through the I/O interface 104 to a display 105 and 
various input devices 106 such as a mouse and keyboard. The 
support circuits can include circuits such as cache, power 
supplies, clock circuits, and a communications bus. The 

10 memory 103 can include random access memory (RAM) , read only 
memory (ROM), disk drive, tape drive, etc., or a combination 
thereof. The present invention can be implemented as a 
routine 107 that is stored in memory 103 and executed by the 
CPU 102 to process the signal from the signal source 108. 

15 As such, the computer system 101 is a general-purpose 

computer system that becomes a specific purpose computer 
system when executing the routine 107 of the present 
invention. 

The computer platform 101 also includes an operating 
20 system and microinstruction code. The various processes and 
functions described herein may either be part of the 
microinstruction code or part of the application program (or 
a combination thereof) , which is executed via the operating 
system. In addition, various other peripheral devices may 
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be connected to the computer platform such as an additional 
data storage device and a printing device. 

It is to be further understood that, because some of 
the constituent system components and method steps depicted 
5 in the accompanying figures may be implemented in software, 
the actual connections between the system components (or the 
process steps) may differ depending upon the manner in which 
the present invention is programmed. Given the teachings of 
the present invention provided herein, one of ordinary skill 
10 in the related art will be able to contemplate these and 
similar implementations or configurations of the present 
invention. 

According to an embodiment of the present invention, a 
term encodes user-interaction within level set methods. 
15 This term is introduced in the form of an evolving shape 
prior and transforms user-edits into level set-based 
propagation constraints. 

For level set representations (see Figure 2) , the level 
set method comprises representing and evolving an evolving 
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curve 




embedding surface 
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0{p;t) = 



0 ,pedR{t) 



-D{{pldR{i))<^ ,pE[n-/?(0] 

where Q is the image domain (bounded) and Z)(p,5i?(^)) is 
the minimum Euclidean distance between the pixel p and the 

curve dR{t) . The level set formulation can be considered as 

an optimization framework. To this end, the approximations 
of Dirac and Heaviside distributions can be defined as: 
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These functions could be used to define contour-based and 
region-based energetic terms in the level set space: 



regional module 



boundary module 



where r and g are region and boundary positive monotonically 
decreasing data-driven functions. The first term [i] is a 
grouping component that accounts for some regional 
properties (modulo the definition of r) of the area defined 
by the evolving curve. The second term [ii] is a 
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combination of a boundary attraction term (modulo the 
definition of jb) and a smoothness component. 

For user-interactive constraints, segmentation 
techniques need local corrections when the visual 
5 information does not support the user-preferred solution. 
User interaction can address this issue. One can consider 
the case of ultrasonic images. The low signal-to-noise 
ratio can lead to segmentation discrepancies. Correcting 
these results can take less time than the complete hand 

10 drawing of the cardiac contours. The user interaction can 
be in the form of points indicated on a level set 
representation to define important elements or mis-detected 
structures. Figure 3a shows an example of a segmentation of 
a heart 301 and user input points 302. Figure 3b shows an 

15 example of the segmentation 301 using the user input points 
as constraints. One of ordinary skill in the art would 
appreciate that other methods of interaction can be 
implemented, such as a touch screen showing a segmented 
image . 

20 Level set methods perform propagation at the pixel 

level and therefore can account for important local 
deformations, but can be sensitive to noise. User 
interactive editing tools can be considered either as local 
or global constraints. The constraints are implemented in 
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an editing procedure to perform correction on the recovered 
solution. To this end, the user is asked to introduce a 
constraint on important or miss-classified parts of the 
segmentation map. 

Two forms of interaction are considered here for 
purposes of explanation. A first interaction uses a one or 
more individual control points for correcting local 
discrepancies, while a second interaction uses a sequence of 
connected points. To derive shape constraints within a 
level set framework, a linear or quadratic interpolation is 
used to convert the user edits into closed structures {e.g., 
shapes) . 

For the construction of a constraint, a quadratic 
interpolation (see Figure 4) determines the value of a 
function at an unknown intermediate point given three data 
points. That is equivalent with fitting a parabola to the 
three data points 

y = Ax^+Bx + C 

v^here A, B, and C are unknowns to be recovered. The data 
points, U.pX-iM-^/'X)' must all lie on the 

curve and are used to determine A, B, and C. The 
determinants can be used to recover these parameters: 
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Constraints can be; (i) independent control points and 
(ii) multiple connected control points provided in a clock- 
wise order. 

For user interaction using a single control point, 
segmentation maps can fail to capture important details due 
to the absence of strong visual support. This limitation 
can be addressed by enforcing the solution to go through 
such a problematic segment. According to an embodiment of 

the present invention, {p) is provided by the user and 
smoothness constraints on the solution are used to correct 
such local discrepancies. 

The solution can be improved locally by replacing a 
small segment of the actual solution with the interactive 

part. Therefore, given a point {p) the curve points that 
lie within a distance d are considered. The use of distance 
maps as embedding function for the level set 
representations, provide a straightforward manner to 
determine these points 
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where S <— 0 . For convex shapes and control points that lie 

on the object, it can be proved that A^^ comprises two 

points for a reasonable small selection for d. Such 
assumption' does not hold for any shape. Therefore, more 
than two points can satisfy the constraint. To introduce 
the interactive segment, only two points of the curve will 
be considered. The points with maximum angular separation 

at (^) are selected: 



(A.pJ-argmaXj.^j|/7,-p. 



A quadratic interpolation is performed between 
i^PnPr^P) interactive segment is determined. Within 

the level set representations, the current position of the 
curve is recovered from the zero-level set of the embedding 
function. The curve points are four-connected (zero- 
crossings on the image plane) , and using a connected 
component rule, recovered in a clock-wise order; 

where is an arbitrary selected point. Towards 
introducing the interactive segment, the segment between 
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and is replaced with the one determined by the quadratic 
interpolation between the control point p and p^ and p^ ; 



Such a shape can be embedded in a level set function 

5 using the Euclidean distance as embedding function; 



1>c(p) = 



0 ,pedUI 
+ D(p,dUl)>0 ,peR„, 
[-D{p,dUl)<0 ,pe[Cl-R„] 



Such a representation encodes the user edits in a 
global fashion using the existing solution in areas where 
user interaction is absent where important deviations from 
10 the constraint should be tolerated. The distance between 
the control point and the image plane is an indicator for 
the importance of the constraint. 

(7^{p) = l + \p-plpen 

Such a measure will be small for the area around the 
15 interactive segment While being significant for the segments 
that are far from the user edits. The same principle can be 
used to account for multiple, independent user edits. 

For user interaction using multiple control points, a 
data-driven solution may not recover a meaningful 
20 segmentation map and to correct errors a large amount of 
local interaction is needed. Global constraints are a 
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different form of user-interaction that can guide the 



segmentation process from the beginning. An exemplary 



scenario is the following: the user provides a minimum 



number of control points in a clock-wise 



order (^p...,^ 



N 



5 that when connected define a closed curve . 

The objective is to recover a global constraint that 
forces the recovered solution to go through the control 
points. The original function can be approximated in a 
piecewise fashion. For any y over the entire domain of a 

10 segment is selected to perform the interpolation. Since the 
shape functions are only defined on each element we can 
approximate y by: 



y{x) = + ^X^k + rX^k., 



where 
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A 




This procedure can convert user interaction into a 



closed structure (e.g., shape) Ry, . The level set 



representation of this structure can be considered to 



enforce the user input. The importance of the constraint 
varies across the image domain and at a given image location 
is inversely proportional to the minimum distance from the 
5 set of control points: 



The user-edits are taken into account when the evolving 
level set representation becomes similar to the one derived 
from the constraint. To this end, the distance between the 



that is equivalent with seeking a curve that goes through 
the user-defined seed points. This is done by minimizing 
15 the distance between the evolving curve and the interactive 
constraint . 

During the model construction, the importance of the 
user-interactive is determined according to the distance 
from the control points. The constraint is enforced more 
20 stringently closer to the user seeds while considering the 
data to guide the segmentation process when there is no 
input from the user. The distance between the constructed 




10 constraint and the evolving representation O is 



considered: 




prior and the control points of the constraint can be used 
to implement such strategy; 

The user interaction is optimally considered when 

5 finding the O that corresponds to the lowest potential of 
the objective function. The calculus of variations within a 
gradient descent method can be used to determine the optimal 
flow that forces the evolving curve to respect the user- 
defined constraints; 

10 o = -2if,(0) ^-S^O)^ ^ 

at (7^ 

V , * V ^ ' 

user-interaction force deflation force 

This flow comprises two terms. The first term evolves 
the curve locally towards the preferred topology as defined 
by the user. The second term is a constant deflation force 
that tends to shrink the curve and consequently minimize the 
15 objective function. Therefore, the second term - the 

deflation component - can be ignored, and only the first 
term is used to account for the user interaction. The flow 
enforces the preferred topology in a qualitative fashion. 

The variability (cT^) of the interaction constraint is used 

20 to downscale the effect of the term in image locations where 
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the user input is not strong. Propagation/segmentation in 
these areas will be data-driven. An image-based term for 
segmentation is defined. Several variational frameworks 
have been proposed for image segmentation. 

For user-interactive geodesic active regions, the 
geodesic active contour can be used for example to perform 
boundary extraction . 



The lowest potential of this functional corresponds to a 
minimal length geodesic curve attracted by the boundaries of 
the structure of interest. Regional/global information can 
improve performance of boundary-based flows that suffer of 
being sensitive to the initial conditions. The central idea 
behind this module is to use the evolving curve to define an 
image partition that is optimal with respect to some 
grouping criterion. The Mumford-Shah framework has been 
used frequently within level set formulations as global 
region-based grouping term using piece-wise constant 
functions; 





monotonically decreasing function. 
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where /Z^,//^ is the mean intensity for the background and 

the object region. The distance from the mean value is 
considered as a region descriptor. The mean values are 
dynamically updated according to the evolving segmentation 
map . 

Integration of the boundary and the region-driven term 
can be considered to perform segmentation, namely the 
geodesic active region model. In th-e absence of noise, 
occlusions, and corrupted visual information, such method 
can be efficient and deal with local deformations. One can 
also integrate the visual terms with the user-interactive 
constraint when available as follows; 

E(o)=^wK ^ (Q>)+wM (0)+w,E^ , (O) 

V / I boundary \ / 2 region \ / 3 interaction \ / 

The calculus of variations as shown earlier for each 
component separately, will provide a curve propagation flow 
that integrates visual support and user interaction. 
Modification of the user preferences can update the 
constraint on the fly. A solution derived from the visual 
terms is recovered and then the user-introduce seeds 
(points) for corrections. Such interaction is then 
converted to propagation force and refines the segmentation 
map towards the user-preferred solution with minimal edits. 
It is important to note that user interaction is introduced 
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in the form of soft-to-hard constraint. The result is a 
segment having an equilibrium between the user edits and the 
solution provided by the data. 

Referring to Figure 5^ a method for boundary based 
image segmentation comprises segmenting an image 501, 
providing a level set representation of the segmentation for 
interaction 502, and providing an interactive edit of the 
level set representation 503. The method comprises 
converting the interactive edit into a propagation 
constraint 504, and determining a segment according to the 
interactive edit and the level set representation 505. 

According to an embodiment of the present invention, a 
framework for user-interaction within the propagation of 
curves uses level set representations. Segmentation 
techniques based on the propagation of curves are very 
popular in image processing and computer vision. Level set 
methods are an emerging formulation to implement these 
techniques with certain strengths as well as some 
limitations. 

Important local deformations as well as topological 
changes can be captured by these techniques. At the same 
time, they refer to an implicit geometry where local 
properties of the evolving curve can be easily determined. 
Although, some of the limitations of these methods - like 
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their inability to account for prior knowledge - have been 
dealt with, no know system or method addresses user 
interaction. According to an embodiment of the present 
invention, interactive editing is converted into propagation 
constraints that force the solution to respect the user 
edits- The construction of such constraints is simple and 
does not require additional computational resources. 

Experimental results demonstrate the potentials of a 
method for interactive segmentation. The user edits can 
correspond to the crosses that appear in the image. To this 
end, a medical example has been considered, the segmentation 
of the left ventricle in polar for ultrasonic images. This 
modality suffers from high signal-to-noise ratio and visual 
support is not sufficient to provide accurate segmentation^ 
results. Global interactive constraints have been used to 
improve segmentation performance of the polar domain. 

Having described embodiments for an interactive level 
set driven image segmentation, it is noted that 
modifications and variations can be made by persons skilled 
in the art in light of the above teachings. It is therefore 
to be understood that changes may be made in the particular 
embodiments of the invention disclosed which are within the 
scope and spirit of the invention as defined by the appended 
claims. Having thus described the invention with the 
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details and particularity required by the patent laws, what 
is claimed and desired protected by Letters Patent is set 
forth in the appended claims. 
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